Systems and methods for calibrating image sensors in wearable apparatuses

ABSTRACT

In one embodiment, a wearable apparatus is provided which may include a holographic film disposed on the wearable apparatus, an eye tracking device including an image sensor and an illuminator, and one or more processors. The processors may be configured to activate the illuminator to illuminate the holographic film and capture, with the image sensor, an image of at least a portion of the holographic film while the holographic film is illuminated. The processors may also be configured to determine a characteristic of the holographic film based on the image and determine, based on the characteristic, a location or an orientation of the image sensor relative to the holographic film. The processors may further be configured to change, based on the location or the orientation of the image sensor relative to the holographic film, at least one calibration parameter of the image sensor.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims the benefit of U.S. Provisional PatentApplication No. 62/631,196, filed Feb. 15, 2018, the entire disclosureof which is hereby incorporated by reference, for all purposes, as iffully set forth herein.

BRIEF SUMMARY OF INVENTION

In one embodiment, an augmented reality, virtual reality, or otherwearable apparatus is provided. The apparatus may include a holographicfilm disposed on the wearable apparatus and an eye tracking deviceincluding an image sensor and an illuminator, and one or moreprocessors. The processors may be configured to activate the illuminatorto illuminate the holographic film and capture, with the image sensor,an image of at least a portion of the holographic film while theholographic film is illuminated. The processors may also be configuredto determine a characteristic of the holographic film based on the imageand determine, based on the characteristic, at least one of a locationor an orientation of the image sensor relative to the holographic film.The processors may further be configured to change, based on at leastone of the location or the orientation of the image sensor relative tothe holographic film, at least one calibration parameter of the imagesensor.

In another embodiment, a method for changing at least one calibrationparameter of an image sensor in an eye tracking device of an augmentedreality, virtual reality, or other wearable apparatus is provided. Themethod may include activating an illuminator of an eye tracking deviceto illuminate a holographic film disposed on a wearable apparatus. Themethod may also include capturing, with an image sensor of the eyetracking device, an image of at least a portion of the holographic filmwhile the holographic film is illuminated. The method may furtherinclude determining a characteristic of the holographic film based onthe image. The method may additionally include determining, based on thecharacteristic, at least one of a location or an orientation of theimage sensor relative to the holographic film. The method may moreoverinclude changing, based on the location or the orientation of the imagesensor relative to the holographic film, at least one calibrationparameter of the image sensor.

In another embodiment, a non-transitory machine-readable medium havinginstructions stored thereon for changing at least one calibrationparameter of an image sensor in an eye tracking device of an augmentedreality, virtual reality, or other wearable apparatus is provided. Theinstructions may be executable by one or more processors to perform amethod. The method may include activating an illuminator of an eyetracking device to illuminate a holographic film disposed on a wearableapparatus. The method may also include capturing, with an image sensorof the eye tracking device, an image of at least a portion of theholographic film while the holographic film is illuminated. The methodmay further include determining a characteristic of the holographic filmbased on the image. The method may additionally include determining,based on the characteristic, at least one of a location or anorientation of the image sensor relative to the holographic film. Themethod may moreover include changing, based on the location or theorientation of the image sensor relative to the holographic film, atleast one calibration parameter of the image sensor.

BRIEF DESCRIPTION OF DRAWINGS

Embodiments of the present invention are described in conjunction withthe appended figures:

FIG. 1 shows an exemplary eye tracking system;

FIG. 2 shows an example image of an eye captured by an image sensor;

FIG. 3 shows a specialized computer system;

FIG. 4 shows an exemplary wearable computing device;

FIG. 5 shows an example components of a virtual reality headset;

FIG. 6 shows an example of a marker on a lens;

FIG. 7 shows an example use of markers on a lens;

FIG. 8 shows a method for updating a calibration parameter;

FIG. 9 shows examples of cornea center-to-lens distances;

FIG. 10 shows an example of image pincushion distortion;

FIG. 11 shows another method for updating a calibration parameter;

FIG. 12 shows an example of an optimized Fresnel lens;

FIG. 13 shows an example of a two-layer Fresnel lens;

FIG. 14 shows an example of manipulating an object in a virtual realityenvironment;

FIG. 15 shows a method for manipulating an object in a virtualenvironment;

FIG. 16 shows an example of manipulating multiple objects in a virtualreality environment;

FIG. 17 shows a method for manipulating multiple objects in a virtualreality environment;

FIG. 18 shows another example of manipulating multiple objects in avirtual reality environment;

FIG. 19 shows another method for manipulating multiple objects in avirtual reality environment;

FIG. 20 shows an example of changing controllers in a virtualenvironment;

FIG. 21 shows a method for changing controllers in a virtualenvironment;

FIG. 22 shows a schematic example of a wearable apparatus demonstratingone embodiment of the invention; and

FIGS. 23A-F shows example holographic films with reflective markersthereon.

In the appended figures, similar components and/or features may have thesame numerical reference label. Further, various components of the sametype may be distinguished by following the reference label by a letterthat distinguishes among the similar components and/or features. If onlythe first numerical reference label is used in the specification, thedescription is applicable to any one of the similar components and/orfeatures having the same first numerical reference label irrespective ofthe letter suffix.

DETAILED DESCRIPTION OF INVENTION

The present application relates to gaze detection systems and methods.In an example, such systems and methods are embodied within virtualreality devices, augmented reality devices, and/or other computingdevices and support user interactions with such computing devices usinggaze information.

Interaction with computing devices is a fundamental action in today'sworld. Computing devices, such as personal computers, are foundthroughout daily life. The systems and methods for interacting with suchdevices defines how they are used, and what they are used for.

Advances in eye tracking technology have made it possible to interactwith a computing device using a person's gaze information. In otherwords, the location on a display the user is gazing at. This informationcan be used for interaction solely, or in combination with acontact-based interaction technique (e.g., using a user input device,such as a keyboard, a mouse, or a touch screen).

Previously proposed interaction techniques using gaze information can befound in U.S. Pat. No. 6,204,828, United States Patent ApplicationPublication 20130169560, U.S. Pat. No. 7,113,170, United States PatentApplication Publication 20140247232, and U.S. Pat. No. 9,619,020. Thefull specification of these patents and applications are hereinincorporated by reference.

Gaze-based interaction techniques are finding their way into wearablecomputing devices, such as virtual reality (VR) headsets and augmentedreality (AR) headsets. These techniques advantageously allow users tocontrol certain aspects of the wearable computing devices (e.g., theuser interface, the settings, the content being presented, etc.).

Nonetheless, given the portability of these types of computing devices,many technical challenges arise. For example, the accuracy of the gazeinformation can degrade for different reasons. A wearable computingdevice is subject to repeated movements, which can change the alignmentof light-processing hardware components. Over time, misalignmentdecreases the accuracy of the information derived from processing themeasured light data. Another reason relates to the dimensions of some ofthese light-processing hardware components (e.g., the used lens). Thedimensions are generally restricted due to the size of the wearablecomputing device. In turn, the restriction can impact how well light isprocessed (e.g., transmitted, reflected, distorted, scattered) by thehardware components.

Furthermore, many applications available on wearable computing devicesinvolve the use of external controllers. Using the external controllersto accurately and intuitively interact with virtual objects impacts theusability, success, and failure of such applications. Integrating thevirtual world with the external controller and the gaze information hasbeen challenging.

Embodiments of the present disclosure relate to improving the accuracyof detecting gaze information and the usability of applications thatrely on external controllers. In the example context of wearablecomputing devices that use gaze-based interaction techniques, theaccuracy can be improved based on a component calibration over timeand/or based on a specific component configuration given the overalldimension restrictions. The component calibration generally includesupdating gaze-related parameters. In a first example, updating agaze-related parameter relies on processing of images to determine theposition and rotation of the camera and lens based on a markerpositioned on the lens. In a second example, updating a gaze-relatedparameter relies on processing of images to determine a distance betweena cornea's center and a lens' center and using this distance tocompensate for distortion (e.g., pincushion distortion). In a thirdexample, updating a gaze-related parameter relies on processing ofimages to determine multiple glint positions and update stereoparameters (e.g., cornea's center and radius, and distance to thecamera). Relevant to the specific component configuration, a Fresnellens is used in an example. A center region of this lens is clear of anyFresnel steps. Instead, the Fresnel steps surround the center region inan annular manner and have varying groove depth and step size. These andother examples are further described herein below. In the interest ofclarity of explanation, components of a gaze detection system aredescribed first, followed by a description of how such components can beimplemented in a wearable portable device. Thereafter, details ofexample component calibrations and example Fresnel lens configurationare described. In the example context of wearable computing devices thatuse gaze-based interaction techniques and external controllers, the gazeof a user on virtual objects in a virtual environment is tracked and avirtual representation of the external controller is presented in thevirtual environment. Upon detecting that the user has gazed on aparticular object for a period of time, the wearable computing deviceselects and locks that object. Given the user's touch area of theexternal controller, the wearable computing device presents a virtualmenu to manipulate the locked virtual object. The options on the menudepend on the user's touch area. In this way, the user is able toaccurately and intuitively manipulate the locked virtual object bymoving his or her finger around the touch area on the externalcontroller.

Eye Tracking

FIG. 1 shows an eye tracking system 100 (which may also be referred toas a gaze tracking system), according to an embodiment. The system 100comprises illuminators 111 and 112 for illuminating the eyes of a user,and an image sensor 113 for capturing images of the eyes of the user.The illuminators 111 and 112 may for example, be light emitting diodesemitting light in the infrared frequency band, or in the near infraredfrequency band. The image sensor 113 may for example be a camera, suchas a complementary metal oxide semiconductor (CMOS) camera or a chargedcoupled device (CCD) camera. The camera is not limited to be an IRcamera or a depth camera or a light-field camera. The shutter mechanismof the image sensor can either be a rolling shutter or a global shutter.

A first illuminator 111 is arranged coaxially with (or close to) theimage sensor 113 so that the image sensor 113 may capture bright pupilimages of the user's eyes. Due to the coaxial arrangement of the firstilluminator 111 and the image sensor 113, light reflected from theretina of an eye returns back out through the pupil towards the imagesensor 113, so that the pupil appears brighter than the iris surroundingit in images where the first illuminator 111 illuminates the eye. Asecond illuminator 112 is arranged non-coaxially with (or further awayfrom) the image sensor 113 for capturing dark pupil images. Due to thenon-coaxial arrangement of the second illuminator 112 and the imagesensor 113, light reflected from the retina of an eye does not reach theimage sensor 113 and the pupil appears darker than the iris surroundingit in images where the second illuminator 112 illuminates the eye. Theilluminators 111 and 112 may for example, take turns to illuminate theeye, so that every first image is a bright pupil image, and every secondimage is a dark pupil image.

The eye tracking system 100 also comprises circuitry 120 (for exampleincluding one or more processors) for processing the images captured bythe image sensor 113. The circuitry 120 may for example, be connected tothe image sensor 113 and the illuminators 111 and 112 via a wired or awireless connection. In another example, circuitry 120 in the form ofone or more processors may be provided in one or more stacked layersbelow the light sensitive surface of the image sensor 113.

FIG. 2 shows an example of an image of an eye 200, captured by the imagesensor 113. The circuitry 120 may for example, employ image processing(such as digital image processing) for extracting features in the image.The circuitry 120 may for example employ pupil center cornea reflection(PCCR) eye tracking to determine where the eye 200 is looking. In PCCReye tracking, the processor 120 estimates the position of the center ofthe pupil 210 and the position of the center of a glint 220 at the eye200. The glint 220 is caused by reflection of light from one of theilluminators 111 and 112. The processor 120 calculates where the user isin space using the glint 220 and where the user's eye 200 is pointingusing the pupil 210. Since there is typically an offset between theoptical center of the eye 200 and the fovea, the processor 120 performscalibration of the fovea offset to be able to determine where the useris looking. The gaze directions obtained from the left eye and from theright eye may then be combined to form a combined estimated gazedirection (or viewing direction). As will be described below, manydifferent factors may affect how the gaze directions for the left andright eyes should be weighted relative to each other when forming thiscombination.

In the embodiment described with reference to FIG. 1, the illuminators111 and 112 are arranged in an eye tracking module 110 placed below adisplay watched by the user. This arrangement serves only as an example.It will be appreciated that more or less any number of illuminators andimage sensors may be employed for eye tracking, and that suchilluminators and image sensors may be distributed in many different waysrelative to displays watched by the user. It will be appreciated thatthe eye tracking scheme described in the present disclosure may, forexample, be employed for remote eye tracking (for example in a personalcomputer, a smart phone, or integrated in a vehicle) or for wearable eyetracking (such as in virtual reality glasses or augmented realityglasses).

FIG. 3 is a block diagram illustrating a specialized computer system 300in which embodiments of the present disclosure may be implemented. Thisexample illustrates a specialized computer system 300 such as may beused, in whole, in part, or with various modifications, to provide thefunctions of components described herein.

Specialized computer system 300 is shown comprising hardware elementsthat may be electrically coupled via a bus 390. The hardware elementsmay include one or more central processing units 310, one or more inputdevices 320 (e.g., a mouse, a keyboard, eye tracking device, etc.), andone or more output devices 330 (e.g., a display device, a printer,etc.). Specialized computer system 300 may also include one or morestorage devices 340. By way of example, storage device(s) 340 may bedisk drives, optical storage devices, solid-state storage devices suchas a random access memory (“RAM”) and/or a read-only memory (“ROM”),which can be programmable, flash-updateable and/or the like.

Specialized computer system 300 may additionally include acomputer-readable storage media reader 350, a communications system 360(e.g., a modem, a network card (wireless or wired), an infra-redcommunication device, Bluetooth™ device, cellular communication device,etc.), and working memory 380, which may include RAM and ROM devices asdescribed above. In some embodiments, specialized computer system 300may also include a processing acceleration unit 370, which can include adigital signal processor, a special-purpose processor and/or the like.

Wearable Computing Device Implementing Eye Tracking

FIG. 4 shows an example of a wearable computing device 400 thatimplements some or all of the above components of an eye tracking systemas described in connection with FIGS. 1-2. The wearable computing device400 can be a VR headset or an AR headset that can be worn by a user. Asillustrated, the wearable computing device 400 includes a set of lenses410, such as Fresnel lenses, a set of cameras 420, a set of hot mirrors430 (e.g., as further illustrated in FIGS. 12-14, the set includes twohot mirrors for each eye in various embodiments), and a set of displays440. The camera 420 can include the image sensors 113 of FIG. 1.Although not shown in FIG. 4, the wearable computing device 400 can alsoinclude a set of illuminators and processing circuitry. These and othercomponents can be integrated within a housing 450 of the wearablecomputing device 400. In this way, upon the user mounting the wearablecomputing device 400 on his or her head, the set of lenses 410 would berelatively close to the user's eyes and the set of displays would berelatively far from the user's eye, and the remaining components may belocated in between. The arrangement of these components allows thedetection of the user's gaze point in three dimensional virtual or realspace.

FIG. 5 shows example components of a VR headset that implements some orall of the above components of an eye tracking system and specializedcomputer system as described in connection with FIGS. 1-5. Asillustrated, the components include a lens cup 510 (also can be referredto as lens tube). In an example, the lens cup 510 is a housing made ofrigid material, such as plastic, and includes two opposite openings 512and 514 along a traversing axis. The shape of the lens cup 510 issubstantially a partial cone-like (or funnel-like) housing along thataxis, with the first opening 512 replacing the conical peak of a fullcone and the second opening 514 at the base. A head-mounted (HMD) lens520 is installed in the first opening 512 and secured with anilluminator cover 530. Light illuminators 540 are disposed within thecover 530 and are external to the lens cup 510. Alternatively, theilluminators 540 may be exposed without the cover 530. The lightilluminators 540 can include a set of light emitting diodes (LEDs) thatemit light in the infrared or near infrared spectrum. A side of the lenscup 510 includes an opening and walls that form a chamber 516. A cameramodule 550 is installed in the chamber and its optical sensor faces theopening of the chamber 516. The components also include a hot mirror560, a VR display 570, and an electronic board 580. The hot mirror 560is installed externally to the lens cup 510 at a location proximate tothe second opening 514. The VR display 570 is installed behind the hotmirror 560 relative to the second opening 514 (e.g., is farther awayfrom the second opening 514). The electronic board 580 includes some orall of the components of the specialized computer system 300 of FIG. 3(e.g., CPU, storage device, computer readable storage media reader,communication system, processing acceleration unit, and working memory).

Hardware Calibration Based on Position and Rotation of Camera and LensDuring Usage

Generally, a wearable computing device, as the one illustrated in FIGS.4-5, is not stable (shaky) because of the user's random movement. Forexample, a sudden drop may cause the movement of internal components,especially the camera and lens. The misalignment of the camera and lensresults in inaccurate eye tracking. To account for the misalignment andimprove the accuracy, calibration can be used.

In existing systems, the calibration would involve manuallydisassembling the wearable computing device and reinstalling thosecomponents into correct positions. And then starting the gazecalibration process from the beginning, which means various parameters(e.g. focal length of the lens, lens diameter, focal length/FOV of thecamera, camera position & orientation, hot mirrors' positions, camera'simaging parameters, illumination parameters and etc.) are needed to beused for the calibration process.

Instead, embodiments of the present disclosure simplify the calibrationprocess significantly. For example, the calibration includes determiningthe misalignment based on the position and/or rotation of the camera andlens and accounting for the position and/or rotation in the computationof the gaze information.

A technical challenge for using this type of calibration is that thereis no direct way to determine the position and rotation of the cameraand lens during usage. The embodiments of the present disclosure use anindirect way by putting and relying on a marker on the lens. Instead ofconsidering the aforementioned parameters, a distance between the camera(e.g.,, a position of the image sensor plane or surface) and the markeron the lens (e.g., the x and y coordinates of the marker) is computedbased on one or more images captured by the camera. By using thedistance value, the eye tracking system may adjust the camera'scalibration parameters (e.g. imaging parameters, focal length,distortion parameters) and illumination parameters. The camera caneither be (i) at least one image sensor with optic arrangement; or (ii)at least one lenseless camera without optic arrangement, but with anultra-thin optical phased array (OPA) which manipulates the incominglight to capture an image without using traditional lenses.

In an example, the embodiments of the present disclosure include storingthe original distance and newly updated distance history information.The storage can be local to the wearable computing device or remote(e.g., retrievable from remote server upon user identification). Thedistance is estimated by using images captured by an eye tracking camera(e.g. IR camera), referred to herein as “camera.” Each one of thoseimages that captured for hardware calibration is stored locally orremoted in non-volatitle (e.g. flash) memory. Whenever the wearablecomputing device is booted up, a new image can be captured as describedherein. The newly captured image can be captured with the stored mostrecent (at least one) image to identify if any relative movement of themarker occurred. If so, at least the newly captured image is furtheranalyzed to perform a calibration. Otherwise, the calibration isskipped.

The marker is at the same side of the camera and can be placed on and/orembedded within one of the lenses; preferably. If there are two or morelenses, one or more markers are placed on and/or embedded within each ofthe lenses because any accidental drop may cause inconsistent movementfor each one of the lenses and its corresponding components. This typeof calibration can also be used in other scenarios. For example, one ormore of the lenses can be moved to compensate for visual artefacts suchas nearsightedness or farsightedness. In this scenario, the opticalsettings could be visually detected in a similar manner (e.g., byrelying on a camera-to-marker distance and/or x and y coordinates of themarker). The marker can be a cross mark, dot pattern (e.g. in-lens LEDilluminator pattern), or a line. The marker can be placed in each one orsome of the optics of the lens group. For example, the marker can beplaced on the one or more layers of the Fresnel lens, and even on thehot mirror. Different markers can be used in different lens elements.For example, a cross may be used as a marker on one lens, a circle onanother lens, and a triangle on a third lens. In this way, the type ofthe marker (e.g., as detected in an image) can be used to associate themarker with a lens.

Since the original relative position and alignment of the camera and thelens is known, once the lens or camera is moved, by using the estimatedposition of the marker, we can “reconstruct” the new relative positionand alignment of the camera and the lens. Accordingly, the eye trackingcamera and illuminators will be recalibrated automatically andimmediately.

Different approaches can be used to trigger the calibration. In oneapproach, the calibration is always ON during the lifetime of theheadset, as long as the headset is powered on or when a user's eyes aredetected by the eye tracking camera. In another approach, a sensor canbe used to trigger calibration. For instance, an accelerometer and/orgyroscope is used for sudden movement detection and then trigger thecalibration process. In yet another approach, the user may use a manualcontrol (e.g. physical button on the VR headset) to initiate therecalibration process. In some or all approaches, a warning feedback(audible or visible or vibration) would be given upon the imperfect eyetracking situation (e.g. in response to the sensor trigger). In afurther example, the trigger is a detection of a user presence (or lackthereof). For example, the calibration may be performed upon detectinguser presence after not having seen the user for a predefined period oftime, such as for 20 seconds.

Prior to the execution of the hardware calibration or runtime of the eyetracking headset, the (factory default) information or data of theposition and alignment of each component inside the eye tracking headsetis stored in a computer readable medium and it is known or inputted intothe system. The components include at least hot mirrors 560,illumination module (530+540), camera module 550, lens cup 510, and VRdisplay 570. The position and alignment data of the components arerepresented by a 3D coordinate system.

In an embodiment, an augmented reality, virtual reality, or otherwearable apparatus is provided. The wearable computing devices of FIGS.4-5 are an example of such apparatus. In an example, the apparatusincludes an eye tracking device and one or more processors. The eyetracking device includes an image sensor, such as a camera, and a firstlens. The first lens includes a first marker. For instance, the firstmarker is on a surface of the first lens or embedded in the first lens.The position of the first marker is within the field of view or field ofvision (FOV) of the camera. The first marker is selected from a groupconsisting of: a cross mark, a dot pattern, a dot, a line, and ageometric shape. Additionally or alternatively, the first markerincludes a lighted element (e.g., one or more LEDs). The apparatus orthe eye tracking device further includes a second lens including asecond marker. The second marker is located at one layer of the lens.Additionally or alternatively, the apparatus further includes a hotmirror including another marker.

The one or more processors are configured (e.g., based oncomputer-readable instructions stored in one or more non-transitorycomputer-readable storage media) to at least receive a first image fromthe image sensor, where the first image shows the first marker;determine a first distance from the image sensor to the first markerbased on the first image; and change at least one calibration parameterof an eye tracking algorithm used with the eye tracking device based onthe first distance.

In an example, changing the at least one calibration parameter isfurther based on the first distance being different than a seconddistance, where the second distance is a previously measured and storeddistance. In an example, changing the at least one calibration parameterbased on the first distance includes determining another distance fromthe image sensor to the first lens based on the first distance and aknown location of the first marker at the first lens, and changing theat least one calibration parameter of the eye tracking algorithm usedwith the eye tracking device based on the second distance. The one ormore processors can be further configured to determine a direction ofthe image sensor relative to the first marker based on the first image.If so, changing the at least one calibration parameter is further basedon the direction.

In a further example, the change of calibration parameters is executedin an event (a) the newly determined distance is different from theprevious determined value; and (b) the adjustment of calibrationparameters is not just based on the newly determined and differentdistance value but also based on the input position and alignment dataof the components.

In an example, if a second marker is included in a second lens, thefirst image shows the second marker. The one or more processors arefurther configured to determine a second distance from the image sensorto the second marker based on the first image; and change at least onecalibration parameter of the eye tracking algorithm based on the seconddistance. In another example, if a second marker is included in a secondlens, the one or more processors are further configured to receive asecond image from the image sensor, where the second image shows thesecond marker; determine a second distance from the image sensor to thesecond marker based on the second image; and change at least onecalibration parameter of the eye tracking algorithm based on the seconddistance. In yet another example, if a second marker is included in ahot mirror, the first image, or a second image received from the imagesensor, shows the second marker. The one or more processors are furtherconfigured to determine a second distance from the image sensor to thesecond marker based on the first image or the second image; and changeat least one calibration parameter of the eye tracking algorithm usedwith the eye tracking device based on the second distance. In thesevarious examples, the at least one calibration parameter is selectedfrom a group consisting of an imagine parameter, a focal length, adistortion parameter, and an illumination parameter.

These and other features are further described in connection with FIGS.6-8. FIG. 6 illustrates an example of a marker on or in a lens. FIG. 7illustrates an example of an image of the marker and the computation ofa distance and a direction based on the image. FIG. 8 illustrates anexample of a flow for updating a calibration parameter.

FIG. 6 illustrates an example of a marker 610 on or in a lens 620,according to certain embodiments of the present disclosure. Asillustrated, the lens 620 can have a geometric shape (e.g., a circle,and ellipse) and have dimensions to fit in a wearable computing device,such as a VR headset or AR headset. For example, the lens 620 is aFresnel lens.

The marker 610 can be attached (e.g., glued) or embedded (e.g., etched,or textured) on an exterior surface of the lens 620 or can be embeddedwithin an interior volume of the lens 620. In an example, the marker 610can have a shape such as a cross mark, a dot pattern, a dot, a line, ageometric shape, and/or can include a lighted element (e.g., anillumination matrix that includes a plurality of LEDs that are arrangedin a predefined pattern on the lens or embedded in the lens). The shapeand/or any pattern conveyed by the shape and/or the lighted element canencode or present information about the marker 610. This information canbe used to compute the distance and direction. For example, theinformation can indicate the size of the marker 610 (which allows toestablish a pixel scale based on an image analysis and the pixel scaleis usable in the computation as further described in connection withFIGS. 7-8). Further, the information can indicate the original locationof the marker 610 on or in the lens 620 (e.g., top of the lens, withinfive millimeters from the outer perimeter of the lens 620), which canalso be used in the computation. The information can also identify themarker 610 or the lens 620 (which is usable to retrieve relevantinformation from memory about the size, location, and or previousdistances and directions).

Different techniques are possible to adjust one or more calibrationparameters based on the position and alignment of components of awearable computing device (e.g., a lens and an image sensor). Generally,existing position and alignment data of the components is available frommemory. An image is captured by the image sensor and shows a marker onthe lens. The distance between the marker and the lens is computed basedon the image. The calibration parameter(s) is adjusted based on theexisting position and alignment data of the internal components and onthe computed distance.

Different techniques are available to compute the relative movementbetween the components of the wearable computing device based on imagedata. In an example, the position and alignment data of each componentis mapped into a three dimensional coordinate system. This system alsoincludes the location of a marker. The relative distance and position ofthe components is known to the system first or stored based on thepreviously determined results. Once the image sensor detects themovement of the marker, the movement can be computed based on theinformation from the image and the information available from the threedimensional coordinate system. Since all the relative positions andalignments are mapped on the same three dimensional coordinate system,the new relative position and alignment of each of the components can beautomatically computed by re-mapping the components to the threedimensional coordinate system based on image data generated by the imagesensor. The difference between the new mapping and the previous mappingindicates the new relative position and alignment, which are the used tochange the calibration parameter(s).

FIG. 7 illustrates an example of an image of a marker and thecomputation of a distance and a direction based on the image, accordingto certain embodiments of the present disclosure. In an example, thedistance is a straight distance between the marker (e.g., a point on asurface of the marker) and a camera 710 (or a point on a surface of thean image sensor of the camera 710). This straight distance indicates theposition of the camera 710 and lens (e.g., a value representing adistance separation between the two). The direction is the direction ofmovement of the marker from a previous location. This directionindicates a rotation of the lens relative to the camera 710 (or viceversa). The determination of the marker movement is not limited to theaforementioned direction calculation. The cause of such movement mayvary due to different manufacture methods of the components.

The number of images used to derive the distance and direction candepend on the information conveyed by the marker and/or the relevantinformation available from memory. In an example, a single imagesuffices when the marker encodes or presents its size and originallocation. Similarly, a single image suffices when the marker identifiesitself or the lens and information about the size and original locationare available from the memory based on the identifier. In theseexamples, the single image shows the marker in its current locationbased on the movement of the camera 710 and/or the lens. Otherwise, twoor more images may be used. One of these images shows the previouslocation, where the previous location corresponds to a previous distanceavailable from the memory. One of the other images shows the currentlocation, for which the distance and direction should be computed. Thedifference between the previous location and the current locationindicates the direction. The use of a single image is described in moredetail herein next, followed by a description of using two images.

Generally, the distance and direction can be defined relative to acoordinate system (e.g., one centered at the center of the image sensorof the camera 710) and can be expressed as a vector (the distance beingthe vector's magnitude and the direction being the vector's direction).A location of the camera (as determined from the image(s)) is mapped tocoordinates in the coordinate system. Further, in the coordinate system,the vector can start at the camera 710 and end at the coordinates of themarker. Geometric reconstruction is applied to compute the vector'sdirection and magnitude based on the image(s).

With a single image, the image shows the marker in its current location.The size of the marker can be identified, directly from the encodedinformation in the image or indirectly from the relevant informationstored in the memory. The size is compared to the number of pixels thatthe marker occupies in the image. The comparison establishes a pixelscale (e.g., if the marker is ten millimeters wide and occupies tenpixels in the image, then each pixel in the image corresponds to onemillimeter).

Thereafter, the distance and direction are computed based on a geometricreconstruction that uses the pixel scale. For instance, a right triangleis formed between the camera 710 and the current location of the markerin the image (the hypotenuse is the location of the camera 710 to thecurrent location), and between the current location and an edge of theimage (e.g., a cathetus). The interior angles and the cathetus aredetermined from the image based on the pixel scale. The hypotenuse(e.g., the distance) is computed using a trigonometric operation.

To compute the direction from the single image, the current location ofmarker in the image is determined and the original location is directlyaccessed (e.g., from the encoded information shown in the image) orindirectly accessed (e.g., from the relevant information in the memory).The two locations are compared and the pixel scale is applied to thedifference between the two locations to derive the direction.

With the use of two or more images, one or more previous locations ofthe marker in one or more previous images are compared to the currentlocation of the marker in the current image. A previous locationcorresponds to a previous known direction and location. The differencewith the current location can be used to generate a difference vector. Apixel scale can be computed as above or from the previous location andthe previous distance. The pixel scale is applied to the differentvector. The magnitude and direction of the difference vector representsthe direction of the marker. Geometric reconstruction is applied toderive the direction from the previous direction and the differencevector.

The use of two images is further illustrated in FIG. 7. The differencebetween the two images is shown as a difference image 720. The markermoves from a previous location 730 (as would be shown in the previousimage) to its current location 740 (as would be shown in the currentimage). The difference between the two establishes a difference vector750. Because the previous distance 760 between the previous location 730and the camera 710 is known, a pixel scale can be derived and applied tothe difference vector 750 to compute its magnitude. The magnitude anddirection indicate the movement of the marker to the new location 740.Geometric reconstruction is then applied to compute a rotational angle780 and the new distance 790 between the marker at the current location740 to the camera 710.

Of course, the marker may not have moved. Instead the camera 710 mayhave moved and a similar approach (a single or multiple images) can beused. Likewise, if both components moved, a similar approach can alsostill be used.

FIG. 8 illustrates an example of a flow for updating a calibrationparameter, according to certain embodiments of the present disclosure.The flow starts at operation 802, where an image is received by acomputer system (e.g., the electronic board 580 of FIG. 5). The imageshows at least one marker. The marker is on or in a lens. The image isgenerated by a camera (e.g., captured by an image sensor of the camera).

At operation 804, the computer system determines a distance and/ordirection from the camera (e.g., from the image sensor) to the markerbased on the image. As described in connection with FIGS. 6-7, variousapproaches are possible. In one approach, a single image can suffice. Inanother approach, a history of the distance and/or location is accessed.That history can include previous images that show the marker atprevious locations. Generally, geometric reconstruction is applied tomeasure the distance and the direction in both approaches.

At operation 806, the computer system determines a distance and/ordirection from the camera (e.g., from the image sensor) to the lensbased on a known location of the marker on the lens. For example, thisdistance and direction is from the center of the image sensor to thecenter of the lens. The known location of the marker identifies a vectorbetween the center of the marker and the center of the lens. This vectoris used to derive, from the distance and direction of operation 804, thedistance and direction between the camera and the lens.

At operation 808, the computer system changes at least one calibrationparameter of an eye tracking algorithm used with the eye tracking devicebased on the distance and/or direction relative to the marker and/orlens as computed at operations 804 and 806. In an example, changing acalibration parameter includes changing one or more of an imagineparameter, a focal length, a distortion parameter, and an illuminationparameter. The change can be specified in a table stored in local memoryof the computer system or remote memory accessible to the computersystem over a data network. The table associates a value of thecalibration parameter with a distance and/or direction between thecamera and another component (e.g., a lens, a hot mirror, etc.). Thevalues in the table can be derived during the design, development, andtesting phase of the wearable computing device under a controlledlaboratory environment or in a trial operational phase.

At operation 810, the computer system determines if other markers areavailable in the image. The presence of another marker can be based onan image analysis or on prior knowledge about the number of markers toexpect. The other marker can be located on or in the same lens, onanother lens, or on a hot mirror. If so, operation 804 is followed. Inthis way, additional distances and/or directions can be computed toupdate calibration parameters. More specifically, if the other marker ison the same lens, the accuracy of the distance and directioncomputations can be increased (e.g., by averaging the two vectors, onefor each marker on the lens). If the other marker is on a different lensor a hot mirror, distance(s) and direction(s) from the camera theretocan be similarly computed and used to further update the relevantcalibration parameter(s) as specified in the table. If no other markersare shown in the image, the flow moves to operation 812.

At operation 812, the computer system determines if other images showingother markers are available. If so, operation 804 is followed. In thisway also, additional distances and/or directions can be computed toupdate calibration parameters. Otherwise, the flow moves to operation814.

At operation 814, the computer system determines if a trigger to repeatthe calibration exists. Various types of triggers are possible. In oneexample type, the calibration is repeated if the wearable computingdevice is powered on or when a user's eyes are detected by its eyetracking system. In another example type, a sensor can be used totrigger calibration. For instance, an accelerometer or gyroscope is usedfor sudden movement detection and then trigger the calibration process.In yet another example type, the user may use a manual control (e.g.physical button on the VR headset) to initiate the recalibrationprocess. If the trigger is received, the flow loops back to operation802 (e.g., operation 802 starts and is repeated based on the triggers).Otherwise, the flow ends.

The distance determination is not limited by using a camera-basedsensing system. A camera-less sensing system is also possible, forexample, a specific type of Micro-Electro-Mechanical Systems (or MEMS)can also be implemented to achieve similar results.

Hardware Calibration Based on Distortion Compensation

Generally, a lens in a wearable computing device, such as the onesillustrated in FIGS. 4-5, suffers from pincushion distortion. Thisdistortion strongly affects the eye image capture directly and affectsthe gaze tracking performance indirectly. Technically, due to the lenspincushion distortion, the captured eye image can have a lot distortionespecially when the eye moves far away from the lens (relative to athreshold value).

In other cases, there might be other types of distortion, usuallyclassified as barrel distortion and mustache distortion. The distortioncan be irregular or follow many patterns. In VR implementation, mostlythe common distortions are radially symmetric. However, this depends onrelative placement and alignment between the camera and lens, the eyeimage may also suffer non-symmetric distortion. In summary, any type ofgeometric distortion is possible and strongly affected by the lensdesign. And the distortion correction model adapts to various kind ofdistortions.

A distortion compensation model is needed to correct captured eye imagesfor gaze tracking. This distortion correction model is dependent on thecornea center-to-lens center distance (shown as D₀ in FIGS. 9 and 10).This means that the distortion model should be adapted to thecornea-to-lens distance at each frame to compensate the eye images forgaze tracking. Further, the distortion model to be used can depend onthe distance to the eye and the position of the eye in x and ycoordinates relative to the lens and the display.

Hence, an updated cornea center-to-lens center value (estimated fromeach frame) is fed into the distortion correction model. To do so, anominal (default) value is set (based on statistical average value) forthe cornea center-to-lens center. The distortion correction model issetup based on the default cornea center-to-lens center. During eyetracking, a cornea center-to-lens center (such as distances D₁, D₂, D₃as shown in FIGS. 9-10) value is estimated automatically at some rate,such as based on each eye image or alternative frame or based on atrigger (image capture operation may be manually adjusted by the user orautomatically according to the headset movement, e.g. whether the userfrequently moves the headset, movement is detected by eye image captureand onboard accelerometer and gyroscope). The cornea center-to-lenscenter is estimated an updated. The newly estimated and updated corneacenter-to-lens center is used to setup a new/updated distortioncorrection model. And the process loops back between setting-up thedistortion correction model and using the new estimates to update thismodel.

If the cornea center-to-lens center distance changed, as shown in FIGS.9-10, the distance varies, and original distance (e.g., D₀) is assumedto be the optimal distance. Generally, the longer the distance (e.g., asD₃), the more pincushion distortion exists.

In an embodiment, an augmented reality, virtual reality, or otherwearable apparatus is provided. The wearable computing devices of FIGS.4-5 are examples of such an apparatus. In an example, the apparatusincludes an eye tracking device and one or more processors. The one ormore processors are configured (e.g., based on computer-readableinstructions stored in one or more non-transitory computer-readablestorage media) to at least receive image data from the image sensor,where the image data corresponds to an image as observed through thelens; determine a level or a pattern of pincushion distortion in theimage based on the image data; and change at least one calibrationparameter of an eye tracking algorithm used with the eye tracking devicebased on the level or the pattern of pincushion distortion.

In an example, changing the at least one calibration parameter based onthe level or the pattern of pincushion distortion includes determining arelative position or relative direction of the image sensor based on alevel or a pattern of pincushion distortion, and changing the at leastone calibration parameter of the eye tracking algorithm used with theeye tracking device based on the relative position or the relativedirection of the image sensor.

In an example, the apparatus further includes a motion sensor. The atleast one processor is further configured to receive a signal from themotion sensor; determine, based on the signal, that the apparatus hasaccelerated or decelerated beyond a threshold amount; and initiate atleast one action in response to determining that the apparatus hasaccelerated or decelerated beyond the threshold amount. The at least oneaction includes at least one selection from a group consisting ofalerting a user of the apparatus, and changing at least one calibrationparameter of the eye tracking algorithm.

These and other features are further described in connection with FIGS.9-11. FIG. 9 illustrates examples of cornea center-to-lens centerdistances. FIG. 10 illustrates an example of an image of the example ofthe pincushion distortion. FIG. 11 illustrates an example of a flow forupdating a calibration parameter based on an image of an eye, where theimage suffers from the pincushion distortion.

FIG. 9 illustrates examples of cornea center-to-lens center distances,according to certain embodiments of the present disclosure. AlthoughFIG. 9 illustrates four distances, shown as D₀, . . . , D₃, any numberof distances is possible depending on the level or pattern of thepincushion distortion and can be generally referred to as D_(i), where“i” is a positive integer. A cornea center represents a center of acornea. The cornea center is determined in three-dimensional space.There are several parameters needed to be considered for determining thecornea center, that includes the position of the camera and the positionat least two illuminators, at least two glint positions (caused by atleast two different illuminators) on the spherical region of the corneaand cornea radius. A lens center represents a center of a lens. A corneacenter-to-lens center distance represents a distance between a center ofa cornea and a center of a lens.

As illustrated, a lens 910 is installed in a wearable computing device(not shown). A nominal value for a distance between the center 912 ofthe lens and the center 922 of a cornea of a human eye ball 920 of auser is defined as distance D₀ 930. This nominal value represents adefault cornea center-to-lens center distance and when the wearablecomputing device is mounted on the user's head and is defined based onstatistical average value across different users wearing this device(either by using data modeling and/or actual data).

When the eye moves away from this default distance, the level and/orpattern of distortion can change. The movement can be caused fordifferent reasons, including changes to the location of the lens 910 inthe wearable computing device (e.g., either intentionally based on auser adjustment, or unintentionally because of the general movement and,sometimes, sudden drops of the wearable computing device by the user).

As the eye 920 moves further away from the center 912 of the lens, thelevel and/or pattern of the pincushion distortion changes. The levelindicates the amount of distortion that would exist in an image of theeye 920, where this image is used for the eye tracking. The patternindicates how the distortion appears in the image.

Hence, at a distance D₁ 940, which represents another value for thecornea center-to-lens center, the level and pattern of the pincushiondistortion is different from the one at the distance D₀ 930. Likewise,at a distance D₂ 950, which represents another value for the corneacenter-to-lens center, the level and pattern of the pincushiondistortion is different from the ones at the distances D₀ 930 and D₁940. Similarly, at a distance D₃ 960, which represents another value forthe cornea center-to-lens center, the level and pattern of thepincushion distortion is different from the ones at the distances D₀930, D₁ 940, D₂ 950. Each of these distances D₁ 940, and D₂ 950, and D₃960 represent a change to the default distance D₀ 930. The larger thedistance, the more distortion exists (e.g., the amount of the distortionincreases and the pattern becomes more pronounced).

FIG. 10 illustrates an example of an image of an example of a pincushiondistortion, according to certain embodiments. Like in FIG. 9, here thepincushion distortion is illustrated in connection with four distancesD₀, . . . , D₃ (that correspond to the distances in FIG. 9). However,any number of distances is possible depending on the level or pattern ofthe pincushion distortion and can be generally referred to as D_(i),where “i” is a positive integer. Although not shown in FIG. 10, theillustrated distortion is that of an image of an eye of a user wearing awearable computing device. And this image is used for eye tracking.

As illustrated, the pincushion pattern changes with the distance D_(i).The larger the distance D_(i), the larger the amount (e.g., level) ofdistortion, resulting in a different pattern. The pattern of thedistortion is illustrated with a particular shape, such as a distortedrectangle (e.g., the vertical sides of the triangle are convex linesinstead of straight lines). The level of distortion is illustrated bythe size, area, and/or perimeter of the particular shape.

Hence, at a distance D₀ 1010, which represents a default value for thecornea center-to-lens center, a default distortion is identifiable froman eye image (shown as the distorted rectangle at the center). At adistance D₁ 1020, which represents another value for the corneacenter-to-lens center, the level and/or pattern of the distortionchanges from the one at the distance D₀ 1010 (the change is shown with alarger distorted rectangle). Likewise, at a distance D₂ 1030, whichrepresents another value for the cornea center-to-lens center, the leveland/or pattern of the distortion changes from the ones at the distancesD₀ 1010 and D₁ 1020 (the change is shown with an even larger distortedrectangle). Similarly, at a distance D₃ 1040, which represents anothervalue for the cornea center-to-lens center, the level and/or pattern ofthe distortion changes from the ones at the distances D₀ 1010, D₁ 1020,and D₂ 1030 (the change is shown with a the largest distortedrectangle).

FIG. 11 illustrates an example of a flow for updating a calibrationparameter based on an image of an eye, where the image suffers from apincushion distortion, according to certain embodiments of the presentdisclosure. The flow starts at operation 1102, where a default corneacenter-to-lens center distance is set by a computer system (e.g., theelectronic board 580 of FIG. 5). This default value can be stored inlocal memory of the computer system and can represent an average corneacenter-to-lens center distance.

At operation 1104, the computer system sets up a distortion correctionmodel based on the default cornea center-to-lens center distance. Forexample, the computer system loads this model from the memory for use inthe eye tracking. In an example, a predictive model is used andassociates levels and/or patterns of pincushion distortion with valuesfor one or more calibration parameters (e.g., a value for an imagineparameter, a focal length, a distortion parameter, and an illuminationparameter, etc.). The associations can include, in the predictive model,a mapping between known images of potential pincushion distortions, thelevels and/or patterns of the potential pincushion distortions, thecornea center-to-lens center distances, and the values of thecalibration parameters. These known images can be stored in thepredictive model itself, or separately in a local memory of the computersystem or a remote memory accessible to the computer system over anetwork. The values can be derived during the design, development, andtesting phase of the wearable computing device under a controlledlaboratory environment or in a trial operational phase. Setting up thedistortion correction model includes, for instance, accessing, from thepredictive model, the values that correspond to a pincushion distortionthat should be observed at the default cornea center-to-lens centerdistance and specifying that these values should be used in the eyetracking to compensate for the distortion.

At operation 1106, the computer system starts the eye tracking.Different triggers are possible. In an example, the tracking starts inresponse to the wearable computing device being turned on, based onsensing that this device has been worn on the user's head, or based on amanual input of the user received at the wearable computing device.Generally, the eye tracking involves analysis of images of at least oneeye of the user. Such eye images can suffer from the pincushiondistortion depending on the cornea center-to-lens center distance.

At operation 1108, the computer system receives image data from an imagesensor (e.g., a camera) of the wearable computing device. This imagedata represents an image of the user's eye and is received as part ofthe eye tracking.

At operation 1110, the computer system determines a level and/or apattern of pincushion distortion in the image based on the image data.In an example, the computer system applies a pattern recognition or animage analysis algorithm to recognize the level and pattern. In anotherexample, the computer system matches the image data to one of the knownimages available from the predictive model, the local memory, or theremote memory. For instance, a best match is identified, and the matchedknown image is used. That known image is associated with the level andpattern of the pincushion distortion in the predictive model.

At operation 1112, the computer system estimates an updated corneacenter-to-lens center distance. In an example, this updated distance isderived from the pattern recognition or image analysis algorithm, if oneis used at operation 1110. In another example, the updated distance isdetermined from the predictive model, where this model associates thelevel and pattern of the pincushion distortion with the updated corneacenter-to-lens center distance. Generally, the distance indicates arelative position or relative direction of the image sensor based on thelevel or the pattern of the pincushion distortion.

At operation 1114, the computer system changes the distortion correctionmodel. In an example, the computer system accesses the values ofcalibration parameters that are relevant to the updated corneacenter-to-lens center distance (or, equivalently, the level and/orpattern of pincushion distortion) and updates the distortion correctionmodel by specifying that these values should be used in connection withthe eye tracking.

At operation 1116, the computer system changes at least one calibrationparameter of an eye tracking algorithm used with the eye tracking devicebased on the level or the pattern of pincushion distortion. For example,the values for the relevant calibration parameters are changed asidentified from the predictive model.

At operation 1118, the computer system determines whether a triggerexists to repeat the change. If so, the flow loops back to operation1108. Otherwise, the flow ends. Various triggers are possible. In oneexample, the change is repeated automatically at a frame interval (e.g.,for each received image or for every other image). In another example,the computer system receives a signal from a motion sensor of thewearable computing device. Based on the signal, the computer systemdetermines that the wearable computing device was moved, and that themovement indicates that the wearable computing device has accelerated ordecelerated beyond a threshold amount. The computer system theninitiates at least one action in response to determining that theapparatus has accelerated or decelerated beyond the threshold amount.That action includes repeating the change and alerting the user. Forinstance, the alert is about the change and can be presented visually ona display and/or audibly on a speaker.

(or, on the pupil)

Optimized Fresnel Lens

Many wearable computing devices use Fresnel lenses. A typicalconfiguration of a Fresnel lens is for a lens that has rings with alarge and constant step size. The problem with these Fresnel steps isthat they are clearly visible in the eye-tracking images (namely eyeimages) because they distort the image locally and they scatter lightfrom the illuminators that can be interpreted as glints. Hence, the eyeimage is useless or difficult to use for gaze determination.

Embodiments of the present disclosure are directed to an optimizedFresnel lens that improves the eye tracking and the user experience. Inan example, the Fresnel steps are smaller than a certain step size. Inthe center region of the Fresnel lens, any Fresnel step can becompletely absent. By not having any Fresnel steps in the center region,stray light is minimized (thereby improving the user experience), theimage quality is improved, and no false glints exist (thereby improvingthe eye tracking). Fresnel steps/rings gradually surround the clearcenter region with increasing groove depth and possibly increasing stepsize.

Hence, the Fresnel lens includes a central region that is flat with nogaps, and variable size Fresnel lens steps outside of the centralregion. Relevant to a small size for a Fresnel lens step, a stepsize/pitch of the Fresnel ring is smaller than the pixel size of thecamera pixels (or if binning is used the size of the combined pixels)divided by the focal length of the camera times the optical distancebetween the camera and the lens times two. A Fresnel ring pitch muchsmaller than this value may generate unnecessary stray light. Outside ofthe central flat region, the groove depth of the Fresnel lens graduallyincreases and possibly the step size increases.

Further, a two layer (or multi layer) Fresnel lens configuration ispossible. Each layer has different lens step and size. The size of thecentral flat region can also be different.

In addition, stray light can be reduced by introducing anti reflection(AR) techniques. Standard anti reflection coatings necessitate complexcoating processes that are expensive and may not scale well with highproduction volumes. Other techniques like Motheye-structure (ME) orplasmaAR® can be more suitable. The resulting surfaces are sensitive tocontaminations so they should not be used on the surface facing the userbut can be used on all other surfaces. Even on the Fresnel surfacesthese AR techniques can be used.

FIG. 12 illustrates an example of an optimized Fresnel lens 1200,according to embodiments of the present disclosure. As illustrated, theFresnel lens 1200 includes a central (or center) region 1210 and Fresnelrings 1220 that surround the central region 1210. The central region1210 is clear of any of such rings and is optically clear such thatlight can pass through that region. Different shapes and sizes of the ofthe central region 1210 are possible depending on the shape and size ofthe Fresnel lens 1200 and its placement relative to other components(e.g., the camera, the area for receiving the user's eye, etc.) in awearable computing device. In an example, the Fresnel lens 1210 has acircular cross section with a diameter in the range of ten to fiftymillimeter. The central region 1210 is also circular and has a diameterin the range of 1% to 25% of the diameter of the Fresnel lens 1200.

In an example, the Fresnel rings 1220 include a number of rings thatannularly surround the central region 1210. The number and sizes ofthese rings 1220 depend on the shapes and sizes of the Fresnel lens 1200and/or central region 1210. Generally, each of the Fresnel rings 1220have a groove depth and a step size. The groove depth and/or the stepsize increases with an increase to the diameter of a Fresnel ring 1220.The groove depth can be in the micrometer range. The step size can be inthe millimeter range.

FIG. 13 illustrates an example of a two-layer Fresnel lens 1300,according to embodiments of the present disclosure. A first layer 1310includes a central region 1312 and a number of Fresnel rings 1314 aroundthe central region 1312. Similarly, a second layer 1320 includes acentral region 1324 and a number of Fresnel rings 1324 around thecentral region 1322. Generally, these central regions 1814 and 1824 arealigned and their center can, but need not be, on the same axis (ahorizontal axis goes through their centers). As illustrated, the size ofthe central regions 1312 and 1324 is different. The size of the Fresnelrings 1314 and 1324 can also be different.

Virtual Environment Control

FIGS. 14-21 illustrate different examples for manipulating virtualobjects in a virtual environment. In the interest of brevity, a virtualobject is referred to herein as an “object.” Generally, objects can berepresented in different layers of the virtual environment. Each objecthas a set of parameters that control how the object is presented(referred to herein as presentation parameters) and manipulationoperations applicable to the object (referred to herein as manipulationparameters).

An external controller is presented in the virtual environment as avirtual controller. Also in the interest of brevity, a virtualcontroller is referred to herein as a “controller” (whereas an externalcontroller is a physical controller). The external controller may haveone or more touch areas, such as a touch-sensitive surface, a tactilebutton, a key switch, etc. These touch areas can be presented as virtualmenus (referred to herein as “menus” in the virtual environment).

Placing a menu in the virtual reality or augmented reality environmentcan be important for user experience. Especially in a complex virtualcontext, having the menu placed in an appropriate position/alignmentmakes the user's operation much more natural and intuitive. In thepresent disclosure, the combination of gaze-based interaction with anobject and controls from the external controller are used to trigger,place and operate the menu.

In an example, the gaze of the user in the virtual environment istracked. If the user gazes at an object for a period of time longer thana time threshold, the object is selected. The selection can lock theobject such that the user can look or gaze away from the object, whilethe object would still be selected and available for manipulation.Thereafter, and depending on the user's hold of a touch area on theexternal controller, a menu specific to that touch area is presented inthe virtual environment. The menu provides options to manipulate theobject according to the parameters of the object. The user operates thetouch area on the external controller to go through and select availablemenu options and manipulate the object in the virtual environment.

In an example, presentation parameters of an object indicate the virtualenvironment layer that the object should be presented in and the lookand feel of the object. Manipulation parameters of the object indicatethe editable visual properties (e.g., the size, shape, color),constraints on moving and placing the object in the virtual environment(e.g., can the object be placed adjacent to another object), andassociations with other objects. The menu corresponding to a touch areaallows edits and manipulations of the object as set in its presentationand manipulation parameters.

Different types of associations can be defined. These associationsimpact how one object can be manipulated in light of other objects inthe virtual environment and, thus, can control how the externalcontroller is usable to manipulate these objects. A “relationshipassociation” is an example and specifies how two objects are related.For instance, a parent-child association is described in connection withFIG. 16. In that figure, a 3D printer is associated with a printedobject, where the printer is a parent and the printed object is a child.In this case, the 3D printer can be manipulated to create the printedobject. Another example of associations is a “manipulation association”that specifies whether a manipulation of an object should carry from orto another object and how such two objects can be manipulated together.For example, a tree is described in connection with FIG. 14. Themanipulation of that tree is independent of other objects in theenvironment based on its manipulation association indicating that thetree is not associated with other objects. In comparison, a card game isillustrated in connection with FIG. 18. In that figure, a card isassociated with slots on a table for receiving the card. Themanipulation of the card can be carried over to a slot such that thecard can be placed in the slot.

FIG. 14 illustrates an example of manipulating an object 1410 in avirtual environment 1400, according to embodiments of the presentdisclosure. A wearable computing device, such as a VR or AR headset asillustrated in FIGS. 4-5, presents the virtual environment 1400. Thewearable computing device is communicatively coupled with an externalcontroller over a data network. The external controller can include anumber of touch areas, each of which provide controls. The data networkcan be a wired or wireless network, including a local area network, apoint-to-point network, a communication bus, or any other data networkssuitable for exchanging control information between the externalcontroller and the wearable computing device. The virtual environment1400 can be a virtual reality environment in the case of a VR headset,or an augmented reality environment in the case of an AR headset.

As illustrated, the virtual environment 1400 presents an object 1410.This object 1410 has a set of parameters that makes it interactive. Inparticular, the user can interact with the object 1410 in the virtualenvironment, where an interaction involves manipulation the object 1410by changing one or more of its properties. The object 1410 is shown as athree dimensional tree.

The wearable computing device presents the virtual environment 1400 on aset of displays and tracks the gaze of the user as he or she views theset of displays and gazes on the displayed virtual environment 1400. Inan example, based on the tracking and a mapping between the set ofdisplays and the virtual environment 1400, the wearable computing devicepresents an indication 1420 of the user's gaze position in the virtualenvironment. For example, the indication 1420 is presented as ageometric shape (e.g., a colored circle) in a layer of the virtualenvironment 1400. In another example, the gaze indication 1420 may beinvisible to the user or may not be displayed. In reality, during suchgaze based interaction, the user already knows what he or she is gazingat. As such, there may not be a need to present the indication 1420.

Further, as the user moves the external controller, the wearablecomputing device detects the movement based on the data communicationwith the external controller over the data network and displays avirtual controller 1430 in the virtual environment 1400. This controller1430 can have the appearance (relative shape and size) to mimic theexternal controller.

If the user holds or is in contact with a touch area of the physicalcontroller, the computing device gets an indication of such a “physicalintegration” in the physical environment from the external controllerand updates the controller 1430 to highlight the corresponding virtualtouch area 1432. As illustrated in FIG. 14, when the user interacts witha physical touch-sensitive area on the external controller, thecorresponding area 1432 is highlighted on the controller 1430 in thevirtual environment.

If the wearable computing device detects the user's gaze on the object1410 (e.g., the three dimensional tree) for a predefined period of time,the wearable computing device may lock the gaze to the object 1410.After that, the user can gaze away from the object 1410 and stillcontrol the object 1410 via the external controller.

Other techniques to lock the object 1410 are possible. For example, agaze region around the object 1410 is predefined. If the wearablecomputing device detects that the user's gaze is within this region andthat, in parallel, receives an indication from the external controllerof a user operation, the wearable computing device locks the object1410. The gaze region can be predefined based on the parameters of theobject. For example, the gaze region is a geometric area that wrapsaround the object 1410 and, thus, is generally bigger than the size ofthe object 1410.

Once the object 1410 is locked, the wearable computing device presents amenu 1440 in the virtual environment. In the example illustration ofFIG. 14, the menu 1440 is positioned in proximity (e.g., adjacent to) tothe object 1410, such that the user can be looking at both the object1410 and menu 1440 while the object 1410 is being manipulated. This menu1440 is in a different layer than that of the object 1410 in the virtualenvironment (as shown in FIG. 14, the menu is in a top layer andincludes in part the object 1810). However, both elements 1410 and 1440can alternatively be in the same layer.

In an example, the menu 1440 itself is in a way a virtual object and hasa set of parameters that control how the look and feel of the menu 1440and the manipulation operations that the menu 1440 supports. Thewearable computing device sets these parameters according to the toucharea on the external controller that the user is interacting with(rather than on the external controller itself). For instance, if thephysical touch area is a circular touch-sensitive area that enablesrotary operations, the menu 1440 is set as a circular menu that alsoenables rotary operations. If the physical touch area is a tactilebutton or a key switch, the menu 1440 is set as a virtual power switch.

In certain examples, the look and feel of the menu 1440 may match thephysical touch area. For example, a circular menu 1440 is presentedwhile the physical touch area is a button or switch like controller.Nonetheless, the menu 1440 is still usable to provide the rotaryoperations but such operations would necessitate different operations inthe physical environment than merely clicking on the button/switch. Forexample, the user may move the external controller in a circular mannerin the physical environment. The wearable computing device may receiveinformation about this movement from the external controller and usethis information to reflect the exerted controls on the object 1410 viathe menu 1440.

Once the menu 1440 is presented, the user can operate the correspondingtouch area (or, the whole controller in case of mismatch as explained inthe previous paragraph) to manipulate the object 1410 according to theparameters of the object 1410 and the supported functionality by themenu 1440 (or, similarly, the physical touch area/controller). Forexample, the user can edit the color, shape, size, location, animation,etc. of the object 1410 in the virtual environment. As such, thewearable computing device receives the relevant control information fromthe external controller, updates the presentation of the menu 1440 toshow that the corresponding controls are being applied, and updates theobject 1410 and its presentation to show the changes to the object 1410.

FIG. 15 illustrates an example flow for manipulating an object in avirtual environment based on gaze information and physical controls onan external controller (e.g., a physical controller), according toembodiments of the present disclosure. The example flow starts atoperation 1502, where a wearable computing device displays the virtualenvironment that includes the object. At operation 1504, the wearablecomputing device displays an indication of the user's gaze position inthe virtual environment. For example, the wearable computing devicetracks and maps the user's gaze to the virtual environment and displaysa geometric shape at the gaze position as mapped in the virtualenvironment.

At operation 1506, the wearable computing device locks the virtualobject. For example, the wearable computing device detects that the gazeposition is on the object for a predefined period of time. In anotherexample, the wearable computing device detects that the gaze position iswithin a predefined gaze region around the object and, in parallel,receives information from the external controller of a user interactionwith the physical controller. In both examples, the wearable computingdevice selects the object and updates its parameters to indicate that ithas been selected and is subject to manipulation.

At operation 1508, the wearable computing device displays a menuassociated with manipulating the object. In an example, the wearablecomputing device presents the menu in proximity of the object in thevirtual environment. This menu can correspond to a physical touch areaon the physical controller with which the user is holding orinteracting. For instance, the wearable computing device receives anidentifier of the touch area from the external controller and accessesfrom memory a virtual model of the external controller and of thephysical touch area. The virtual model can define the shape, size, lookand feel, and functionalities of a corresponding virtual touch area. Thewearable computing device presents this virtual touch area as the menu.

At operation 1510, the wearable computing device manipulates the object.In an example, the wearable computing device receives controlinformation from the external controller based on user interaction withthe physical touch area. Because the object is locked, the wearablecomputing device presents a virtual representation of the userinteraction on the menu, manipulates the object based on the controlinformation, and presents in the virtual environment the changes to theobject as being manipulated.

FIG. 16 illustrates an example of manipulating multiple objects in avirtual environment, according to embodiments of the present disclosure.Here, two objects are associated with each other: a three dimensional(3D) printer 1610 and a printed object 1630 (e.g., a box). The 3Dprinter 1610 can be manipulated in the virtual environment to print theprinted object 1630. In other words, the creating of the second object1630 depends on the manipulation of the first object 1610.

As illustrated in the top-left corner of FIG. 16, the wearable computingdevice detects the user's gaze over the object 1610 (the virtual 3Dprinter). Based on parameters of this object, a guide 1620 about usingthis object is presented. The guide 1620 presents information in thevirtual environment about operating the external controller to interactwith the object 1610.

In an example, the wearable computing device sets the guide 1620 toreplicate at least the shape of the virtual controller (that, in turn,replicates the shape of the external controller). The operationinformation can be shown relative to the replicated shape. Asillustrated in FIG. 16, the guide 1620 informs the user to “press” theblue touch area.

Once the user presses the corresponding touch area on the externalcontroller, the wearable computing device receives an indication of thispress therefrom. Because of the association between the virtual 3Dprinter and the virtual printed object, the wearable computing devicegenerates and presents the object 1630 (e.g., the virtual printedobject) in the object 1610 (e.g., in the virtual 3D printer), asillustrated in the top-right corner of FIG. 16.

At that point, the user can gaze at the object 1630 to lock it.Alternatively, the wearable computing device locks it automatically(without a specific gaze) based on the generation of the object 1630since the user already pressed the blue touch area in order to print theprinted object in the virtual environment. Once locked, the object 1630can be manipulated.

Accordingly, the wearable computing device presents a menu to manipulatethe object 1630. In an example, the wearable computing device receives,from the external controller, an indication of a user interaction with aphysical touch area of the external controller. In response, thewearable computing device invokes and presents the menu. Thepresentation can be gradual such that the menu transitions from aninitial state to a stable state (e.g., the menu transitions forwardtowards the user in the virtual environment).

The bottom-left corner of FIG. 16 shows the menu in transition from theinitial state (shown as in-transition menu 1640). In transition, themenu is displayed in a first tilted angle. The wearable computing deviceupdates the presentation of the menu such that it gradually movestowards the user's front field-of-view. During the movement of the menu,the tilted angle is adjusted to a second tilted angle until the end andstable placement in front of the user in the virtual environment. Thebottom-right corner of FIG. 16 shows the menu in the stable state at thesecond tilted angle (shown as stable menu 1650).

Once fully displayed, the menu includes multiple components that can bearranged on one or more layers of the virtual environment. Some of thesecomponents can be objects that can be locked and manipulated. Forexample, based on detecting gaze position over the “width” component,the wearable computing device locks this component for manipulation. Themanipulation can occur via user interaction with a touch area of theexternal controller.

FIG. 17 illustrates an example flow for manipulating multiple objects ina virtual environment based on gaze information and physical controls onan external controller (e.g., a physical controller), according toembodiments of the present disclosure. The example flow starts atoperation 1702, where a wearable computing device detects gaze positionon a first object (e.g., a virtual 3D printer) in the virtualenvironment. At operation 1704, the wearable computing device displays aguide associated with using the first object. In an example, the guideis displayed based on parameters of the first object and its associationwith a second object (e.g., a virtual printed object). The guide showsinformation about using the external controller to manipulate the firstobject. For example, the guide informs the user about pressing physicalcontrols (e.g., interacting with a physical touch area) to generate thesecond object.

At operation 1706, the wearable computing device displays the secondobject in the virtual environment based on interaction information. Inan example, the interaction information includes control informationprovided from the external controller based on the user pressing thephysical controls in response to the guide's information. The wearablecomputing device receives the interaction and generates and displays thesecond object accordingly.

At operation 1708, the wearable computing device locks the secondobject. In an example, the second object is locked automatically (e.g.,without additional user input or user gaze) based on the second objectbeing generated in response to receiving the interaction information.

At operation 1710, the wearable computing device displays a menuassociated with manipulating the second object. In an example, the menucorresponds to a physical touch area of the external controller andfacilitates manipulation of the second object based on user interactionswith the physical touch area. Some or all components of the menu arealso objects that can be locked based on detection of gaze of them and,once locked, can be manipulated based on user interaction with thephysical touch area on the external controller.

FIG. 18 illustrates another example of manipulating multiple objects ina virtual environment, according to embodiments of the presentdisclosure. Here, two objects are associated with each other: a card1810 and a slot 1850 of a table. The card 1810 is a first object thathas parameters specifying that it can be moved in the virtualenvironment and placed in the slot 1850. The slot is a second objectthat has parameters specifying that it is static in the virtualenvironment and that it can be a destination object for other objects(e.g., can receive the card 1810). In other words, the manipulation ofthe first object 1810 can involve a manipulation of the second object1850.

As illustrated in the top-left corner of FIG. 18, the wearable computingdevice detects the user's gaze 1840 over the object 1810 (the card) fora predefined period of time. Based on parameters of this object, theobject 1810 is selected. The wearable computing device shows theselection 1830 by highlighting the object 1810.

Next, the wearable computing device receives information from theexternal controller of a user interaction with a touch area of theexternal controller (e.g., a press on a button). The wearable computingdevice determines that this information corresponds to a hold of theobject 1810 and according updates the presentation of the object 1810 toshow that it is being held in the virtual environment.

Once held, the wearable computing device detects that the user's gazemoved away from the object 1810 (e.g., the card) to the destinationobject 1850 (e.g., the slot on the table for receiving the card). Thefar-right corner of FIG. 18 shows the updated gaze as element 1810 beingpositioned on the destination object 1850.

After the user's gaze at the destination object 1850, the wearablecomputing device may allow the user to gaze away and operate theexternal controller to move the object 1810 to the destination object1850. Upon receiving information from the external controller of a userinteraction with the touch area of the external controller (e.g.,another button press), the determination is made that the object 1810should be placed in the destination object 1850 and, accordingly,updates the virtual environment to show that the object 1810 was droppedin the destination object 1850, as shown with element 1860 of thebottom-center part of FIG. 18.

FIG. 19 illustrates another example of a flow for manipulating multipleobjects in a virtual environment, according to embodiments of thepresent disclosure. Here, two objects are associated with each othersuch that a first object can be moved in proximity to (e.g., be placedin, on, under, etc.) a second object in the virtual environment.

The example flow starts at operation 1902, where a wearable computingdevice detects gaze position on the first object in the virtualenvironment for a predefined period of time. Accordingly, the wearablecomputing device locks the first object.

At operation 1904, the wearable computing device detects a userinteraction for holding the first object in the virtual environment. Forexample, the computing device receives information from the externalcontroller about the user interaction, where the user interaction iswith a touch area of the external controller and indicates the hold.Accordingly, the wearable computing device updates parameters of thefirst object and its presentation in the virtual environment to reflectthe hold.

At operation 1906, the wearable computing device detects gaze positionon the second object in the virtual environment for a predefined periodof time. Accordingly, the wearable computing device locks the secondobject.

At operation 1908, the wearable computing device detects a userinteraction for placing the first object onto the second object. Forexample, the computing device receives information from the externalcontroller about the user interaction, where this interaction is withthe touch area of the controller and indicates the placement.

At operation 1910, the wearable computing device places the first objectonto the second object based on the detected user. For example, thewearable computing device updates parameters of the first object and itspresentation in the virtual environment to reflect the placement.

FIG. 20 illustrates an example of changing controllers in a virtualenvironment, according to embodiments of the present disclosure. Here,two objects are associated with each other: a current controller 2010and an available controller 2020. These two objects are mutuallyexclusive from being usable in the virtual environment. In other words,the user is permitted to only be using one of them at any time in thevirtual environment.

In an example, the user may be using an external controller and thewearable computing device can present a corresponding controller in thevirtual environment. The user may change the external controller toanother one. The wearable computing device would change the controllerin the virtual environment accordingly. Alternatively, the user may beusing a generic external controller that can be mapped to differentvirtual models stored in local memory, or accessible via remote memoryto the wearable computing device. The wearable computing device presentsthese different virtual models as available controllers in the virtualenvironment and allows the user to switch between them.

As illustrated in the top-left corner of FIG. 20, the wearable computingdevice displays the current controller 2010 that the user is operatingin the virtual environment and other available controller(s) 2020 forselection. As illustrated in the top-right corner of FIG. 20, to switchfrom the current controller 2010 to an available controller 2020, theuser operates the external controller such that, in the virtualenvironment, the current controller 2010 is moved towards the positionof the available controller 2020. For example, the wearable computingdevice receives information from the external controller indicating auser interaction with a touch area of the external controller or theexternal controller itself, where the user interaction is for moving thecurrent controller 2010 towards available controller 2020. Based on thisinformation, the wearable computing device updates the parameters andthe presentation of the current controller 2010 to show this movement inthe virtual environment.

To switch to the available controller 2020, the user may need to punchor tap the available controller. Here also, the user operates theexternal controller to perform this operation in the virtual environmentand the wearable computing device receives the relevant information fromthe external controller and updates the virtual environment. As shown inthe top-right corner of FIG. 20, the wearable computing devicehighlights the virtual punch or tap 2030 by changing the visualproperties of the available controller 2420 (e.g., by setting its colorto bright or to flash).

Once the available controller 2020 is selected (shown as selectedcontroller 2040 in the bottom-center part of FIG. 20), the currentcontroller 2010 is replaced with the selected controller 2040. In anexample, the current controller 2010 is an object presented in a toplayer of the virtual environment. The available controller 2020 is alsoanother object presented, prior to the selection, in a lower layer ofthe virtual environment. Based on the selection, the wearable computingdevice dismisses (e.g., removes from the presentation) the objectrepresenting the current controller 2010 from the top layer and adds theobject representing the available controller 2020 in this top layer.Alternatively, rather than removing and adding, the wearable computingdevice changes the parameters of the object in the top layer from thoseof the current controller 2010 to the parameters of the availablecontroller 2020.

FIG. 21 illustrates an example of a flow for changing controllers in avirtual environment, according to embodiments of the present disclosure.The example flow starts at operation 2102, where a wearable computingdevice detects movement of a first controller towards a secondcontroller in the virtual environment. For example, the computing devicereceives information from the external controller about a userinteraction with a touch area of the external controller or the externalcontroller itself, where the user interaction indicates the movement.Accordingly, the wearable computing device updates parameters of thefirst controller and its presentation in the virtual environment toreflect the movement.

At operation 2104, the wearable computing device detects contact (e.g.,virtual contact) of the first controller with the second controller inthe virtual environment. This contact corresponds to a virtual punch ortap. For example, the computing device receives information from theexternal controller about a user interaction with the touch area of theexternal controller, where the user interaction indicates the virtualcontact.

At operation 2106, the wearable computing device selects the secondcontroller based on the virtual contact. At operation 2108, the wearablecomputing device replaces the first controller with the secondcontroller in the virtual environment.

FIG. 22 shows a schematic example of a wearable apparatus 2200demonstrating an embodiment of the invention. In some embodiments,wearable apparatus 2200 may be an augmented reality or virtual realityheadset. In this embodiment, wearable apparatus 2200 is shown as onetype of a virtual reality headset.

Wearable apparatus 2200 may include an eye tracking device 2210 whichincludes at least at least one image sensor 2213 and at least oneilluminator 2216 (referred to herein as image sensor 2213 andilluminator 2216, respectively). Eye tracking device 2210 may alsoinclude one or more processors 2219, or a connection to one or moreprocessors remote from eye tracking device 2210 and/or apparatus 2200,but performing the same and/or similar functions (all referred to hereinas processor 2219).

Illuminator 2216 may transmit one or more wavelengths of light, andimage sensor 2213 may be configured or selected to be responsive tocorresponding wavelengths. In some embodiments, the light provided byilluminator 2216 may be in the infrared wavelength, and consequentlyimage sensor 2213 may be response to infrared wavelength light.

Eye tracking device 2210 may be configured to selectively activateilluminator 2216 to illuminate at least one eye 2220 of the user ofwearable apparatus 2200. Processor 2219 may analyze images of theilluminated eye 2220 and determine a gaze direction of eye 2220 basedtherefrom.

Wearable apparatus 2200 may also include at least one holographic film2230 disposed at some location on wearable apparatus 2200. In theexample shown, holographic film 2230 is shown in three possiblelocations on wearable apparatus 2200. In a first possible location,holographic film 2230 a is shown on a lens 2240 of wearable apparatus2200. In a second possible location, holographic film 2230 b is shown ona display 2250 of wearable apparatus 2200. In a third possible location,holographic film 2230 c is shown on a stand-alone (non-lens) portion2260 of wearable apparatus 2200. In practice only one holographic film2230 may be present. In some embodiments, multiple distinct holographicfilms 2230 may be present. In various embodiments, image sensor 2213 maybe positioned at an angle (θ) relative to normal of the gaze ray'sincidence with holographic film 2230. In some embodiments, θ may bebetween about 5 degrees and about 75 degrees.

Additionally, note that the orientation and placement of eye trackingdevice 2210 (and consequently image sensor 2213 and/or illuminator2216), as well as holographic film 2230 may be modified from thisexample in a given embodiment. For example, illuminator 2216 may bepositioned relatively remote from image sensor 2213, and more directlyaddress eye 2220, with light reflected from eye being reflected again byholographic film 2230 toward image sensor 2213. Furthermore, indifferent apparatuses than virtual reality headsets, holographic film2230 may be disposed on other portions of such apparatuses (i.e., alight-guide of an augmented reality headset).

Holographic film 2230 may be any sort of film which allows forholographic patterns and images to be presented thereon. Examplematerials for holographic film 2230 include polyester, polypropylene,nylon, and/or other polymers. In some embodiments, a surface reflectionhologram is employed, where patterns and images may be embossed thereon,possibly via creation of tiny grooves cut into the film at variousangles and in different shapes. This allows for the presentation ofpatterns and images which may have a three-dimensional effect. In otherembodiments, a volumetric reflection hologram may be employed, where thehologram is created by exposing holographic film with light thatselectively changes the refractive index of the film.

Reflections of patterns and images may be specular or diffuse. Specularreflections proceed from holographic film 2230 in a single direction,while diffuse reflections proceed from holographic film 2230 inmultiple/many directions. Holographic film 2230 with either or both typeof reflectivity may be employed as will be discussed below.

When holographic film 2230 with specular reflections is employed,holographic film 2230 must be disposed on portions of apparatus 2200where the reflective angles relative to image sensor 2213 andilluminator 2216 are such that image sensor 2213 will indeed be able todetect reflections on holographic film 2230 from illuminator 2216. Whenholographic film 2230 with diffuse reflections is employed, a broaderrange of relative positions of image sensor 2213, illuminator 2216, andholographic film 2230 are possible.

In an activity outside the normal process of using eye tracking device2220 to determine a gaze direction of eye 2220, processor 2219 may alsobe configured to activate illuminator 2216 to illuminate holographicfilm 2230 and capture, with image sensor 2213, an image of at least aportion of holographic film 2230 while holographic film 2230 isilluminated. Processor 2219 may then determine a characteristic ofholographic film 2230 based on the image. For example, processor 2219may determine the size and relative positions of different reflectiveportions of holographic film 2230.

Based on the determined characteristics of holographic film 2230,processor 2219 may then determine a location, an orientation, and/orother characteristic of image sensor 2213 relative to holographic film2230 (or a some critical sub-component of image sensor 2213; forexample, a lens, a charge-coupled device (CCD), active pixel sensor,etc.). Processor 2219 may then be configured to change at least onecalibration parameter of image sensor 2213, based on at least one of thelocation, orientation, or other characteristic.

Alternatively, processor 2219 may send data regarding the location,orientation, and/or other characteristic to image sensor 2213 for use byimage sensor 2213 to change at least one calibration parameter ofitself. In a further alternative, processor 2219 may use data regardingthe location, orientation, and/or other characteristic to calibrate analgorithm used by processor 2219 to determine gaze direction using imagesensor 2213 and illuminator 2216. The calibration parameters adjustedcould include, merely by way of example, an imaging parameter, a focallength, a distortion parameter, an illumination parameter, anorientation of the image sensor relative to the hologram, a field ofview, a distance to the hologram, and/or an angle of the image sensorrelative to holographic film 2230.

Holographic film 2230 may have varied reflective characteristics, eachof which may be sufficient to implement embodiments of the inventiondescribed. In order for an image of reflective light from holographicfilm 2230 to be sufficient to determine a location and/or orientation ofimage sensor 2213 therefrom, characteristics in at least two orthogonalaxes must be present. In some embodiments, a single feature having twodimensional characteristics will be provided. In other embodiments, twoor more features may be present which have combined two dimensionalcharacteristics.

FIGS. 23A-F show a variety of holographic films with differentcharacteristics which could be used to implement embodiments of theinvention. Other variations are possible. Hatched portions of thesefigures represent a first reflective portion thereof, while non-hatchedportions represent a second reflective portion. However, the reflectionangle of the first reflective portion may be different than thereflection angle of the second reflective portion. Alternatively,hatched or non-hatched portions of the figures could representnon-reflective areas of the film. I.e., a hole or cavity in theholographic film/mirror, also useful for calibration based on thecharacteristics thereof (size, location, etc.). In some embodiments, thenon-hatched portions could reflect images of the eye for gaze detection,while the hatched portions could provide a specular reflection towardimage sensor 2213 for use in calibration as discussed herein. While morecomplex and larger markers on the holographic films can provide moreinformation and lead to more exactly determined locations/orientationsof image sensor 2213, smaller markers will reduce the possibility ofimage sensor 2213 being saturated by reflective light, likely makingdata therefrom less or not usable.

FIG. 23A shows a holographic film with a marker that has two dimensionalcharacteristics. This marker could be presented diffusely or specularlydepending on placement of illuminator 2216 and image sensor 2213.Diffuse markers may be made in any way known in the art, or that becomesknown in the art in the future.

FIG. 23B shows an alternative marker that also has two dimensionalcharacteristics. This marker could also be presented diffusely orspecularly depending on placement of illuminator 2216 and image sensor2213.

FIG. 23C shows another marker whereby at least three holographic spheresare presented. Reflections of illumination of the spheres provide threegenerally point locations in a two dimensional layout. Overallreflections from the spheres would be spread in different directions dueto the rounded nature of the holographic spheres. Larger spheres wouldprovide more reflective light, but less accuracy than smaller sphereswhich would provide less reflective light, but with increased accuracy.Other three dimensional holographic markers may also be employed.

FIG. 23D shows another possible marker where both the dimensions of theilluminated area, as well as the implicit dimensions of thenon-illuminated area could function as characteristics useable by thesystems and methods of the invention to determine location and/ororientation of the image sensor 2213. This marker could also bepresented diffusely or specularly depending on placement of illuminator2216 and image sensor 2213.

FIG. 23E shows another marker whereby the overall dimensions of theentirely reflective holographic film provide the necessary twodimensional characteristics. This marker could also be presenteddiffusely or specularly depending on placement of illuminator 2216 andimage sensor 2213.

FIG. 23F shows another marker having two single dimension (or nearlysingle dimension) marks whereby the length of the marks, as well astheir relative placement to each other provide the necessary twodimensional characteristics. This marker could also be presenteddiffusely or specularly depending on placement of illuminator 2216 andimage sensor 2213.

The disclosure has now been described in detail for the purposes ofclarity and understanding. However, it will be appreciated that certainchanges and modifications may be practiced within the scope of theappended claims.

The above description provides exemplary embodiments only, and is notintended to limit the scope, applicability or configuration of thedisclosure. Rather, the above description of the exemplary embodimentswill provide those skilled in the art with an enabling description forimplementing one or more exemplary embodiments. It being understood thatvarious changes may be made in the function and arrangement of elementswithout departing from the spirit and scope of the disclosure as setforth herein.

For example, any detail discussed with regard to one embodiment may ormay not be present in all contemplated versions of that embodiment.Likewise, any detail discussed with regard to one embodiment may or maynot be present in all contemplated versions of other embodimentsdiscussed herein. Finally, the absence of discussion of any detail withregard to embodiment herein shall be an implicit recognition that suchdetail may or may not be present in any version of any embodimentdiscussed herein.

Specific details are given in the following description to provide athorough understanding of the embodiments. However, it will beunderstood by one of ordinary skill in the art that the embodiments maybe practiced without these specific details. For example, circuits,systems, networks, processes, and other elements in the disclosure maybe shown as components in block diagram form in order not to obscure theembodiments in unnecessary detail. In other instances, well-knowncircuits, processes, algorithms, structures, and techniques may be shownwithout unnecessary detail in order to avoid obscuring the embodiments.

Also, it is noted that individual embodiments may be described as aprocess which is depicted as a flowchart, a flow diagram, a data flowdiagram, a structure diagram, or a block diagram. Although a flowchartmay describe the operations as a sequential process, many of theoperations can be performed in parallel or concurrently. In addition,the order of the operations may be re-arranged. A process may beterminated when its operations are completed, but could have additionalsteps not discussed or included in a figure. Furthermore, not alloperations in any particularly described process may occur in allembodiments. A process may correspond to a method, a function, aprocedure, a subroutine, a subprogram, etc. When a process correspondsto a function, its termination corresponds to a return of the functionto the calling function or the main function.

The term “machine-readable medium” includes, but is not limited totransitory and non-transitory, portable or fixed storage devices,optical storage devices, wireless channels and various other mediumscapable of storing, containing or carrying instruction(s) and/or data. Acode segment or machine-executable instructions may represent aprocedure, a function, a subprogram, a program, a routine, a subroutine,a module, a software package, a class, or any combination ofinstructions, data structures, or program statements. A code segment maybe coupled to another code segment or a hardware circuit by passingand/or receiving information, data, arguments, parameters, or memorycontents. Information, arguments, parameters, data, etc. may be passed,forwarded, or transmitted via any suitable means including memorysharing, message passing, token passing, network transmission, etc.

Furthermore, embodiments of the disclosure may be implemented, at leastin part, either manually or automatically. Manual or automaticimplementations may be executed, or at least assisted, through the useof machines, hardware, software, firmware, middleware, microcode,hardware description languages, or any combination thereof. Whenimplemented in software, firmware, middleware or microcode, the programcode or code segments to perform the necessary tasks may be stored in amachine readable medium. A processor or processors may perform thenecessary tasks.

As used herein, the phrase “a first thing based on a second thing,” andthe like, may mean that the first thing is based solely on the secondthing, or that the first thing is based on the second thing as well asone or more additional things.

What is claimed is:
 1. An augmented reality, virtual reality, or otherwearable apparatus comprising: a holographic film disposed on a wearableapparatus; an eye tracking device comprising an image sensor and anilluminator; and one or more processors configured to at least: activatethe illuminator to illuminate the holographic film; capture, with theimage sensor, an image of at least a portion of the holographic filmwhile the holographic film is illuminated; determine a characteristic ofthe holographic film based on the image; determine, based on thecharacteristic, at least one of a location or an orientation of theimage sensor relative to the holographic film; and change, based on atleast one of the location or the orientation of the image sensorrelative to the holographic film, at least one calibration parameter ofthe image sensor.
 2. The augmented reality, virtual reality, or otherwearable apparatus of claim 1, wherein the image comprises: a reflectionof light off the holographic film.
 3. The augmented reality, virtualreality, or other wearable apparatus of claim 2, wherein the reflectioncomprises: a diffuse reflection from a feature of the holographic film.4. The augmented reality, virtual reality, or other wearable apparatusof claim 2, wherein the reflection comprises: a specular reflection of afeature of the holographic film.
 5. The augmented reality, virtualreality, or other wearable apparatus of claim 1, wherein the imagecomprises: a diffuse reflection of light surrounding an area having noreflection of light.
 6. The augmented reality, virtual reality, or otherwearable apparatus of claim 1, wherein the image comprises: a diffusereflection of light bounded by an edge of the holographic film.
 7. Theaugmented reality, virtual reality, or other wearable apparatus of claim1, wherein determining the location or the orientation of the imagesensor relative to the holographic film comprises: determining thelocation or the orientation of the image sensor in relation to threeorthogonal axes.
 8. The augmented reality, virtual reality, or otherwearable apparatus of claim 7, wherein the holographic film comprises:three characteristics disposed in two axes in the holographic film whichspecularly reflect light toward the image sensor.
 9. The augmentedreality, virtual reality, or other wearable apparatus of claim 7,wherein the holographic film comprises: only one characteristic whichreflects light, wherein the characteristic is two-dimensional.
 10. Theaugmented reality, virtual reality, or other wearable apparatus of claim7, wherein the holographic film comprises: at least one characteristicwhich does not reflect light, wherein at least some portion of the atleast one characteristic is bounded by reflective portions of theholographic film, and wherein the at least one characteristic istwo-dimensional.
 11. A method for changing at least one calibrationparameter of an image sensor in an eye tracking device of an augmentedreality, virtual reality, or other wearable apparatus, the methodcomprising: activating an illuminator of an eye tracking device toilluminate a holographic film disposed on a wearable apparatus;capturing, with an image sensor of the eye tracking device, an image ofat least a portion of the holographic film while the holographic film isilluminated; determining a characteristic of the holographic film basedon the image; determining, based on the characteristic, at least one ofa location or an orientation of the image sensor relative to theholographic film; and changing, based on the location or the orientationof the image sensor relative to the holographic film, at least onecalibration parameter of the image sensor.
 12. The method for changingat least one calibration parameter of an image sensor in an eye trackingdevice of an augmented reality, virtual reality, or other wearableapparatus of claim 11, wherein the image comprises: a reflection oflight off the holographic film, and wherein the reflection comprises adiffuse reflection from a feature of the holographic film or a specularreflection of a feature of the holographic film.
 13. The method forchanging at least one calibration parameter of an image sensor in an eyetracking device of an augmented reality, virtual reality, or otherwearable apparatus claim 11, wherein the image comprises: a diffusereflection of light surrounding an area having no reflection of light.14. The method for changing at least one calibration parameter of animage sensor in an eye tracking device of an augmented reality, virtualreality, or other wearable apparatus of claim 11, wherein the imagecomprises: a diffuse reflection of light bounded by an edge of theholographic film.
 15. The method for changing at least one calibrationparameter of an image sensor in an eye tracking device of an augmentedreality, virtual reality, or other wearable apparatus of claim 11,wherein determining the location or the orientation of the image sensorrelative to the holographic film comprises: determining the location orthe orientation of the image sensor in relation to three orthogonalaxes.
 16. A non-transitory machine-readable medium having instructionsstored thereon for changing at least one calibration parameter of animage sensor in an eye tracking device of an augmented reality, virtualreality, or other wearable apparatus, wherein the instructions areexecutable by one or more processors to at least: activate anilluminator of an eye tracking device to illuminate a holographic filmdisposed on a wearable apparatus; capture, with an image sensor of theeye tracking device, an image of at least a portion of the holographicfilm while the holographic film is illuminated; determine acharacteristic of the holographic film based on the image; determine,based on the characteristic, at least one of a location or anorientation of the image sensor relative to the holographic film; andchange, based on the location or the orientation of the image sensorrelative to the holographic film, at least one calibration parameter ofthe image sensor.
 17. The non-transitory machine-readable medium ofclaim 16, wherein the image comprises: a reflection of light off theholographic film, and wherein the reflection comprises a diffusereflection from a feature of the holographic film or a specularreflection of a feature of the holographic film.
 18. The non-transitorymachine-readable medium of claim 16, wherein the image comprises: adiffuse reflection of light surrounding an area having no reflection oflight.
 19. The non-transitory machine-readable medium of claim 16,wherein the image comprises: a diffuse reflection of light bounded by anedge of the holographic film.
 20. The non-transitory machine-readablemedium of claim 16, wherein determining the location or the orientationof the image sensor relative to the holographic film comprises:determining the location or the orientation of the image sensor inrelation to three orthogonal axes.